An Interactive Virtual Reality Platform for Studying Embodied Social Interaction

We present an interactive virtual reality platform for studying the role of embodied social interaction in the context of language learning. The virtual environment consists of virtual objects, a virtual table, and most importantly, a set of virtual students with different social-cognitive skills. Real users are asked to serve as language teachers and teach virtual learners object names. They can interact with virtual learners via gazing, pointing at and moving virtual objects as well as speech acts. Since both the virtual environment (what users see) and the virtual humans (whom users interact with) are controlled (pre-programmed), this provides a unique opportunity to study how real teachers perceive different social signals generated by virtual learners and how they adjust their behaviors accordingly. One primary result is that real people feel comfortable to interact with virtual humans in the virtual environment and treat them as social partners. Moreover, the platform allows us to record real people’s multimodal behavioral data and analyze the data across individual participants to extract shared behavioral patterns. Overall, this work demonstrates the usefulness of virtual reality technologies in studying both human-human and human-machine social interactions.